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January  27,  1998 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  (COMPTROLLER) 

DIRECTOR,  DEFENSE  FINANCE  AND  ACCOUNTING 
SERVICE 


SUBJECT:  Audit  Report  on  the  Defense  Finance  and  Accounting  Service  Acquisition 
Program  for  the  Electronic  Document  Management  R'ogram 
(Report  No.  98-057) 


We  are  providing  this  audit  report  for  information  and  use.  The  Director, 
Defense  Finance  and  Accounting  Service,  requested  that  we  review  the  implementation 
of  the  Electronic  Document  Management  Program  through  the  integrated  product  team 
process.  This  report  is  the  first  in  a  series  of  reports  to  be  issued  on  the  Defense 
Finance  and  Accounting  Service’s  acquisition  strategy  for  the  Electronic  Document 
Management  Program. 

Because  this  report  contains  no  recommendations,  written  comments  were  not 
required.  However,  we  received  verbal  comments  from  Defense  Finance  and 
Accounting  Service  personnel  updating  the  status  of  Increment  1,  Vendor  Pay, 
life-cycle  documentation  and  a  copy  of  the  December  16,  1997,  memorandum  from  the 
Major  Automated  Information  System  Review  Council  granting  the  Program  Milestone 
III  approval  (Appendix  F).  We  have  incorporated  this  information  into  the  final  report. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the  audit 
should  be  directed  to  Ms.  Kim  Caprio,  Audit  Program  Director,  at  (703)  604-9139 
(DSN  664-9139)  or  Mr.  Eric  L.  Lewis,  Acting  Audit  Project  Manager,  at 
(703)  604-9144  (DSN  664-9144).  See  Appendix  G  for  the  report  distribution.  The 
audit  team  members  are  listed  inside  the  back  cover. 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  98-057 

(Project  No.  7FG-0029) 


January  27,  1998 


Defense  Finance  and  Accounting  Service  Acquisition  Program 
for  the  Electronic  Document  Management  Program 


Executive  Summary 


Introduction.  The  Defense  Finance  and  Accounting  Service  (DFAS)  Electronic 
Document  Management  Program  (the  Program)  will  standardize  document  distribution, 
tracking,  and  storage.  The  Program  is  expected  to  improve  processing  time,  reporting 
accuracy,  and  customer  service,  resulting  in  reduced  personnel  costs.  On  January  5, 
1996,  the  Deputy  Assistant  Secretary  of  Defense  (Command,  Control, 

Communications,  and  Intelligence)  designated  the  Program  as  a  major  automated 
information  system  subject  to  the  review  of  the  Major  Automated  Information  System 
Review  Council.  The  DFAS  Electronic  Document  Management  Program  Office 
identified  three  functional  areas  and  is  developing  them  on  an  incremental  basis; 

o  Increment  1,  Vendor  Pay,  which  pays  commercial  invoices; 

o  Increment  2,  Contract  Pay,  which  pays  large  contracts;  and 

o  Increment  3,  Garnishment,  which  collects  court  ordered  withholdings. 

This  report  provides  the  results  of  our  review  of  the  Increment  1,  Vendor  Pay, 
life-cycle  documentation.  The  Vendor  Pay  increment  will  standardize  and  automate  the 
document  workflow  for  payments  of  commercial  invoices  at  five  initial  DFAS  centers 
or  operating  locations.  In  fiscal  year  1996,  under  the  current  system,  DoD  paid 
approximately  $290  million  in  personnel  costs  to  process  13.7  million  invoices. 
Increment  1,  if  implemented  at  the  17  locations  proposed,  will  have  an  estimated 
life-cycle  cost  of  $404  million  for  fiscal  years  1995  through  2009.  The  estimated 
life-cycle  costs  for  the  first  phase  of  five  locations  is  $100  million  for  fiscal  years  1995 
through  2004.  The  DFAS  expects  that  reductions  in  personnel  costs  made  possible  by 
the  Program  will  yield  a  return  on  investment  but  has  not  yet  developed  an  estimate  of 
the  amount. 

This  audit  report  is  the  first  in  a  series  of  reports  on  the  DoD  acquisition  strategy  for 
the  DFAS  Electronic  Document  Management  Program  and  provides  the  results  of  our 
review  of  the  Increment  1,  Vendor  Pay,  life-cycle  documentation.  The  Director, 
DFAS,  requested  that  we  review  the  implementation  of  the  Program  through  the 
integrated  product  team  process  and  provide  input  during  the  acquisition  process. 


Audit  Objectives.  The  overall  audit  objective  was  to  assess  the  DFAS  development 
and  implementation  of  the  Program.  Specifically,  we  reviewed  the  acquisition 
documentation  for  Increment  1,  Vendor  Pay.  We  also  reviewed  management  controls 
related  to  the  objective. 

Audit  Results.  The  integrated  product  teams  identified  cost,  funding,  and  testing 
concerns  that  needed  to  be  resolved  before  a  deployment  decision  could  be 
recommended.  The  Program  Office  provided  a  cost  reconciliation  document,  funding 
information,  and  a  schedule  for  testing  to  minimize  the  concerns  of  the  integrated 
product  teams.  DFAS  developed  the  required  life-cycle  documentation  and 
subsequently  received  a  Milestone  III  deployment  decision,  for  Increment  1,  Vendor 
Pay,  of  the  Program  on  December  16,  1997.  See  Part  I  of  this  report  for  further 
details.  Management  controls  were  adequate  in  that  no  material  management  control 
weakness  was  identified.  See  Appendix  A  for  a  discussion  of  the  management  control 
program. 

During  the  course  of  the  audit  we  suggested  that  management: 

o  reconcile  the  benefits  of  other  DoD  vendor  pay  initiatives  with  the  benefits 
expected  from  the  Electronic  Document  management  solution; 

o  determine  which  sites  will  be  fielded,  how  much  workload  will  be  automated, 
and  update  the  cost  and  test  plans  based  on  those  determinations; 

o  provide  cost  and  performance  metrics  approved  by  the  Program  Analysis  and 
Evaluation  personnel,  limit  fielding  to  a  number  of  sites  necessary  to  capture  feedback 
data,  conduct  an  in-process  review  subsequent  to  the  milestone  decision,  and  obtain 
adequate  funding  before  fielding  to  additional  sites; 

o  establish  a  baseline  for  each  site  prior  to  implementation  and  adjust  the  total 
Program  baseline  accordingly; 

o  establish  criteria  to  determine  when  a  baseline  is  breached; 

0  request  a  review  be  made  by  Operational  Test  and  Evaluation  personnel  prior 
to  the  implementation  of  new  sites  or  changes  in  size  or  workflow  at  existing  sites;  and 

o  request  Test  Systems  Engineering  and  Evaluation  personnel  review  stress 
testing  to  determine  the  maximum  throughput  and  the  level  of  growth  the  System  can 
handle. 

Because  management  took  responsive  action  to  suggestions  made  during  the  review  of 
Milestone  III  documentation,  this  report  contains  no  recommendations.  However,  the 
Program  Office  should  continue  to  ensure  that  the  agreed-upon  actions  are  completed 
within  the  timeframes  established  by  the  Major  Automated  Information  System  Review 
Council  and  that  all  IPT  members  are  informed  of  the  status  of  ongoing  actions. 
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Part  I  -  Audit  Results 


Introduction 


This  audit  report  is  the  first  in  a  series  of  reports  on  the  DoD  acquisition 
strategy  for  the  Defense  Finance  and  Accounting  Service  (DFAS)  Program  and 
provides  the  results  of  our  review  of  the  Increment  1,  Vendor  Pay.  We 
previously  issued  two  reports  concerning  other  Electronic  Document 
Management  (EDM)  Program  issues.  (See  Appendix  B  for  details  of  those 
reports.) 


Audit  Background 


Electronic  Document  Management  Program.  The  EDM  Program  (the 
Program)  evolved  from  the  DFAS  Document  Imaging  Program.  The  Imaging 
Program  began  as  a  management  improvement  initiative  in  May  1991  that 
included  business  process  improvement  efforts,  workflow  applications,  and 
electronic  document  management  systems.  The  Program  initiative  reviewed 
during  this  audit  will  reduce  the  paper  flow  and  automate  the  payment  process, 
potentially  reducing  cycle  time  by  20  percent  for  commercial  invoices. 

On  October  11,  1994,  DFAS  designated  the  Imaging  Program  as  an  official 
acquisition  program.  When  the  DFAS  Electronic  Document  Management 
Program  Office  (the  Program  Office)  updated  the  mission  needs  statement  in 
February  1997,  several  of  the  same  functional  areas  were  still  being  considered 
for  the  EDM  solution.  The  Program  Office  identified  three  functional  areas  or 
“increments,”  in  which  to  begin  implementing  the  EDM  solution:  Vendor  Pay, 
Contract  Pay,  and  Garnishment.  DFAS  treats  each  functional  area  as  a  separate 
increment.  Each  increment  has  its  own  life-cycle  documentation  such  as  the 
operational  requirements  document,  test  and  evaluation  master  plan,  cost 
analysis  requirements  description,  and  life-cycle  cost  estimate.  Increments  must 
identify  benefits  and  be  cost  effective  for  a  milestone  approval.  (A  milestone  is 
the  decision  point  that  separates  the  phases  of  an  acquisition  program.  For 
definitions  of  the  individual  acquisition  documents  and  milestones,  see 
Appendix  C.) 

Increment  1.  Increment  1,  Vendor  Pay,  provides  the  prototype  solution  to  be 
the  basis  for  the  remaining  increments.  For  example.  Increment  2,  Contract 
Pay,  will  expand  on  the  core  vendor  pay  system  by  adding  workflows,  such  as 
contract  close-out,  that  are  unique  to  large  contracts  associated  with  the 
Mechanization  of  Contract  Administration  Services  system.  The  vendor  pay 
prototype  was  implemented  at  the  Omaha,  Nebraska,  operating  location  and  is 
currently  operational.  Increment  1  was  to  be  implemented  at  17  DFAS  centers 
or  operating  locations  to  support  the  vendor  pay  function.  The  DFAS  vendor 
pay  function  processed  over  13.7  million  invoices  in  fiscal  year  1996.  DFAS 
estimates  the  life-cycle  costs  to  fully  deploy  17  sites  would  be  $404  million  for 
fiscal  years  1995  through  2009  in  fiscal  year  1996  dollars.  The  estimated 
life-cycle  cost  for  the  first  phase  of  Increment  1 ,  covering  five  sites,  is 
$100  million  for  fiscal  years  1995  through  2004.  Full  operational  capability  for 
the  first  five  sites  is  expected  in  October  1999. 
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Review  Process.  The  acquisition  of  DoD  information  systems,  regardless  of 
size,  undergo  a  review  process.  A  review  is  held  at  each  milestone  to  gain 
approval  to  enter  the  next  phase  of  the  life  cycle.  To  obtain  a  milestone 
decision,  the  program  office  develops  the  life-cycle  management  documentation. 

When  the  Program  began,  the  oversight  body  was  DFAS’s  Information 
Management  Executive  Board.  In  October  1994,  the  DFAS  Information 
Management  Executive  Board  granted  the  Imaging  Program  Milestone  0  and  I 
decisions  and  allowed  the  Program  to  proceed  with  development.  A 
Milestone  0  decision  gives  approval  to  conduct  concept  studies,  while  a 
Milestone  I  decision  gives  approval  to  begin  a  new  acquisition  program.  In 
support  of  the  milestone  decisions,  the  Program  Office  had  prepared  a  mission 
needs  stotement,  an  analysis  of  alternatives,  a  requirements  analysis,  an 
acquisition  plan,  and  a  program  management  plan. 

On  January  5,  1996,  the  Deputy  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  designated  the  Program  as  a  major 
automated  information  system  subject  to  the  review  of  the  Major  Automated 
Information  System  Review  Council  (MAISRC).  Consequently,  the  Program 
must  be  managed  under  the  criteria  of  DoD  5000. 2-R,  “Mandatory  Procedure 
,  for  Major  Defense  Acquisition  Program  (MDAPs)  and  Major  Automated 
Information  Systems  (MAIS)  Acquisition  Program,”  March  15,  1996.  The 
Regulation  requires  program  offices  to  develop  life-cycle  documentation  to 
justify  and  document  their  programs.  DFAS’s  Information  Management 
Executive  Board  had  already  granted  the  Program  a  Milestone  0  and  I  decision, 
so  the  MAISRC  honored  those  decisions  and  identified  the  next  milestone  for 
Increment  1  to  be  a  combination  Milestones  II  and  III. 

Because  this  was  the  first  formal  MAISRC  examination  of  the  Program,  some 
of  the  documents  generally  required  for  earlier  milestones  were  updated  and 
reapproved.  The  mission  needs  statement  is  required  for  the  Milestone  0 
decision;  however,  the  Program  Office  revised  the  original  mission  needs 
statement  and  obtained  the  Under  Secretary  of  Defense  (Comptroller)  approval 
in  February  1997.  The  mission  needs  statement  is  needed  for  a  Milestone  0 
decision  because  it  documents  deficiencies  in  current  capabilities  and 
opportunities  to  provide  new  capabilities  expressed  in  broad  operational  terms. 

Audit  Objectives 

The  overall  audit  objective  was  to  assess  DFAS’s  development  and 
implementation  of  the  Program.  Specifically,  we  reviewed  the  acquisition 
documentation  for  Increment  1,  Vendor  Pay.  We  also  reviewed  management 
controls  related  to  the  objective.  See  Appendix  A  for  a  discussion  of  the  audit 
scope  and  methodology  and  Appendix  B  for  prior  coverage  related  to  the  audit 
objective. 
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Acquisition  Documentation  Review  for 
Increment  1  of  the  Program 

The  Defense  Finance  and  Accounting  Service’s  Electronic  Document 
Management  Program  Office  had  requested  its  first  Milestone  III 
Deployment  Approval  decision  fi'om  the  MAISRC  on  Increment  1,  Vendor 
Pay,  of  the  Program.  The  Program  Office  prepared  for  the  decision  by 
using  the  integrated  product  team  approach  to  conduct  testing,  prepare 
life-cycle  documentation,  identify  and  resolve  issues,  and  to  make  sound 
and  timely  suggestions  to  facilitate  Program  decisionmaking.  The 
integrated  product  teams  identified  cost,  funding,  and  testing  concerns  that 
needed  to  be  resolved  before  a  deployment  decision  could  be 
recommended.  As  a  result,  the  Program  Office  provided  a  cost 
reconciliation  document,  funding  information,  and  a  schedule  for  testing 
and  subsequently  received  on  December  16,  1997,  Milestone  III  approval 
to  deploy  Increment  1  to  five  sites. 


Designation  as  a  MAISRC  System 


The  EDM  Program  was  designated  as  a  major  automated  information  system  in 
January  1996  and  was  seeking  a  Milestone  III,  Deployment  Approval,  decision 
from  the  MAISRC.  This  decision  would  allow  the  Program  Office  to  procure 
additional  equipment  and  deploy  the  system  to  the  identified  locations  in 
accordance  with  the  acquisition  decision  memorandum.  To  ensure  that 
deployed  systems  satisfy  user  requirements,  the  Secretary  of  Defense  had 
directed  the  use  of  integrated  product  teams  (IPTs)  to  facilitate  the  oversight  and 
review  of  the  life-cycle  management  documentation. 

Life-Cycle  Management  Documentation.  DoD  5000.2-R  r^uires  the 
program  office  to  prepare  an  acquisition  strategy  and  an  acquisition  program 
baseline  for  a  Milestone  III  decision.  The  mission  needs  statement,  the  analysis 
of  alternatives,  the  operational  requirements  document,  the  test  and  evaluation 
master  plan,  a  life-cycle  cost  estimate,  and  a  cost  analysis  requirements 
description  are  other  life-cycle  documentation  that  support  the  acquisition 
program  baseline  and  decisionmaking.  These  documents  provide  the 
requirements,  affordability,  and  performance  data  needed  to  make  acquisition 
decisions  at  an  acceptable  level  of  risk. 

Integrated  Product  Teams.  The  IPTs  consist  of  qualified  team 
members  from  the  appropriate  functional  disciplines  who  are  empowered  to 
make  commitments  for  the  organizations  or  functional  areas  that  they  represent. 
Working  with  a  team  leader,  IPTs  build  successful  and  balanced  programs, 
identify  and  resolve  issues,  and  make  sound  and  timely  recommendations  to 
facilitate  decisionmaking. 

The  Program  Office  established  six  IPTs  (Cost,  Test,  Training,  Vendor  Pay 
Solution,  Contract  Pay  Solution,  and  Security)  to  review  life-cycle 
documentation  as  it  was  being  developed.  This  aided  in  the  exchange  of 
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Acquisition  Documentation  Review  for  Increment  1  of  the  Program 


information  and  understanding  of  the  Program  among  the  decisionmakers.  IPT 
members  included  MAISRC  action  officers  (representatives  of  the  MAISRC 
members),  who  reported  their  conclusions  and  concerns  to  their  respective 
MAISRC  members.  The  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  action  officer  ensured  that  the  MAISRC 
Chairman  was  aware  of  IPT  member  conclusions  and  concerns. 

This  audit  focused  on  the  Cost  and  the  Test  IPTs.  The  Cost  IPT  focused  on  the 
issues  affecting  funding,  cost,  and  benefits.  These  issues  are  addressed  in  such 
life-cycle  documentation  as  the  mission  needs  statement,  cost  analysis 
requirements  description,  life-cycle  cost  estimate,  and  acquisition  program 
baseline.  The  Test  IPT  focused  on  issues  relating  to  the  operational 
requirements  document  and  the  test  and  evaluation  master  plan.  These  two 
documents  are  closely  related  because  the  testing  plan  is  based  on  the 
requirements  established  in  the  operational  requirements  document.  The  Test 
IPT  also  addressed  Year  2000  compliance  and  security  issues.  Subsequently,  , 
the  Security  IPT  was  formed  in  August  1997. 


Deploying  Increment  1  of  the  Program 


The  Program  Office  had  requested  its  first  Milestone  III  Deployment  Approval 
decision  from  the  Major  Automated  Information  Systems  Review  Council  on 
Increment  1 ,  Vendor  Pay.  The  life-cycle  documentation  supporting  the 
Program  was  in  good  order.  When  shortcomings  were  identified  in  the 
documentation,  the  IPT  was  able  to  resolve  most  issues.  The  Program  Office 
stated  that  documentation  alone  could  never  address  all  the  information  needed 
to  make  a  fully  informed  Milestone  III  decision.  Therefore,  the  IPT  process 
was  utilized  to  discuss  issues  and  to  avoid  time-consuming  “paperwork”  drills. 
As  a  result,  the  IPTs  completed  efforts  on  many  of  the  life-cycle  documents  and 
has  developed  an  effective  process  to  resolve  open  issues. 

Ongoing  Efforts,  IPT  efforts  to  resolve  Increment  1  life-cycle  costs,  test  and 
evaluation,  and  acquisition  baseline  issues  are  ongoing. 

Life-Cycle  Costs.  The  Program’s  life-cycle  cost  estimate  and  the  cost 
analysis  requirements  description  were  prepared  as  required;  however,  the  IPT 
informed  the  Program  Office  that  additional  support  was  needed  for  some  of  the 
cost  assumptions.  The  life-cycle  cost  estimate  should  consider  all  the  costs  to 
determine  whether  the  solution  is  affordable  and  the  cost  analysis  requirements 
description  provides  detailed  information  supporting  the  life-cycle  cost  estimate. 
The  additional  support  would  allow  decisionmakers  to  make  fully  informed 
decisions  about  the  Program’s  cost  assumptions.  Specifically,  the  cost 
documentation  was  prepared  based  on  the  assumption  that  the  EDM  solution 
would  be  implemented  at  17  processing  locations  and  did  not  take  into 
consideration  the  effects  of  other  ongoing  initiatives  on  DoD  vendor  payment 
processing,  such  as  DFAS  consolidation  efforts  and  the  use  of  credit  cards. 

Processing  Locations.  DFAS  is  currently  undergoing  a  review 
of  its  entire  infrastructure,  which  includes  analyzing  where  vendor  pay  is 
processed.  As  a  result,  fewer  than  17  sites  may  be  needed  to  support  vendor 
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pay.  Because  these  studies  are  not  complete,  the  Program  Office  has  initially 
requested  approval  to  deploy  the  Program  to  only  five  sites.  Additional  sites 
Avill  be  evaluated  on  a  case-by-case  basis. 

Credit  Cards.  Under  the  Federal  Acquisition  Streamlining  Act 
of  1994,  DoD  bills  under  $2,500  can  be  paid  by  credit  card  without 
documentation.  This  initiative  accounts  for  up  to  12  percent  of  all  invoices  and 
could  account  for  more  if  the  initiative  is  expanded.  To  assure  that  the  EDM  is 
not  over-fielded,  the  Program  Office  stated  that  it  would  not  field  beyond  the 
five  sites  unless  analyses  show  that  further  savings  were  possible. 

Test  and  Evaluation.  The  test  and  evaluation  master  plan  is  a 
high-level  blueprint  for  testing  the  selected  solution  to  determine  whether  it  will 
satisfy  mission  needs.  The  Program’s  test  and  evaluation  master  plan  was 
prepared  and  approved  as  required;  however,  the  IPT  reported  that  stress  testing 
(to  determine  the  effects  of  significant  workload  changes),  system  security 
testing,  and  Year  2000  testing  remained  to  be  accomplished. 

Stress  testing  is  usually  accomplished  as  part  of  developmental  testing  to 
determine  the  amount  of  processing  the  system  can  handle  before  failure.  The 
stress  test  information  is  essential  because  it  allows  the  developer  to  determine 
whether  the  proposed  system  configuration  will  cost-effectively  meet  the  current 
and  expected  mission  need.  However,  the  Test  IPT  was  not  formed  until  after 
the  Program  had  undergone  unit  acceptance  testing.  The  Test  IPT  did  not 
require  the  Program  Office  to  change  its  development  schedule  so  that  a  stress 
test  could  be  run,  rather  the  IPT  members  assessed  all  available  test  information 
to  determine  whether  a  formal  stress  test  was  necessary.  After  the  assessment, 
the  IPT  found  that  a  stress  test  would  provide  essential  information  on  the 
systems  processing  and  growth  capabilities.  The  Program  Office  agreed  and 
scheduled  the  testing  for  February  1998. 

Acquisition  Program  Baseline.  Program  Analysis  and  Evaluation 
personnel  stated  that  the  acquisition  program  baseline  is  the  most  important 
life-cycle  document  during  a  Milestone  III  decision  process.  Specifically,  the 
acquisition  program  baseline  should  present  the  most  likely  cost,  schedule,  and 
performance  characteristics  of  the  selected  solution  and  serve  as  the  basis  for  the 
acquisition  decision  memorandum.  The  Program  Office  prepares  the  acquisition 
program  baseline  and  submits  it  to  the  MAISRC  to  show  how  the  program 
office  proposes  to  manage  the  acquisition  program.  The  MAISRC  will  issue  an 
acquisition  decision  memorandum  that  either  accepts  or  modifies  the  acquisition 
program  baseline.  The  acquisition  decision  memorandum  becomes  the  contract 
between  the  MAISRC  and  the  program  office.  If  a  program’s  cost,  schedule, 
and  performance  adversely  deviates  from  the  acquisition  decision  memorandum 
the  program  manager  must  provide  an  explanation  for  the  discrepancy  and 
develop  a  plan  to  correct  the  discrepancy.  The  EDM  Program’s  acquisition 
program  baseline  needed  more  specific  information  in  order  to  establish 
supportable  baselines.  In  particular,  the  Program’s  life-cycle  cost 
documentation  had  to  be  reconciled  with  the  component  cost  analysis/sufficiency 
review  to  establish  a  baseline.  The  acquisition  program  baseline  had  a  life-cycle 
cost  estimate  of  $104  million  for  fiscal  years  1995  through  2004  for  five  sites; 
that  figure  changed  to  $100  million  after  the  reconciliation.  The  MAISRC 
provided  the  following  direction  to  accomplish  the  reconciliation  of  the  cost 
estimate. 
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Cost  Estimate  Reconciliation  and  Site  Funding.  The 
component  cost  analysis/sufficiency  review  and  the  Program  Office  estimate 
were  to  be  reconciled  and  the  Program  Office  was  to  ensure  that  the  first  five 
sites  were  fully  funded.  The  MAISRC  gave  the  Program  Office  60  days  from 
August  1,  1997,  to  resolve  these  issues.  If  these  issues  were  not  resolved  within 
60  days,  the  Program  Office  had  to  brief  the  Overarching  IPT  before 
preceding  with  further  predeployment  activities.  The  DFAS  Resource 
Management  Deputate  completed  the  cost  reconciliation  on  September  15,  1997, 
and  provided  the  MAISRC  with  the  appropriate  funding  documentation. 

Acquisition  Decision  Memorandum  Directions.  While  the  cost 
documentation  issues  were  being  resolved,  the  MAISRC  established  the  steps 
necessary  to  establish  the  Program’s  baselines.  The  MAISRC  established  these 
steps  based  on  the  work  of  the  IPT.  The  Program  Office  will  carry  out  the 
proposed  acquisition  decision  memorandum  directions  by  performing  additional 
test  and  evaluation  and  security  and  performance  measurement.  Additionally, 
the  Program  Office  will  continue  the  development  of  Increment  2  (Contract 
Pay).  Increment  2  will  provide  Program  services  for  the  Mechanization  of 
Contract  Administration  Services  system  and  will  have  a  separate  Milestone  111 
decision.  The  final  acquisition  decision  memorandum  was  signed  on 
December  16,  1997,  which  allows  the  Program  Office  to  field  the  Increment  1 
of  the  Program  at  up  to  five  sites.  (See  Appendix  F  for  a  copy  of  the 
memorandum) 

Completed  Efforts.  The  IPTs  have  completed  efforts  and  resolved  Increment  1 
issues  regarding  mission  needs,  analysis  of  alternatives,  and  operational 
requirements. 

Mission  Needs  Statement.  The  mission  needs  statement  demonstrated 
that  a  business  area  deficiency  existed  that  cannot  be  satisfied  with  just  a  change 
in  business  practices.  The  Program’s  mission  needs  statement  was  prepared  and 
approved  as  required.  However,  the  IPT  members  (including  the  Program 
Office)  recognized  that  the  document  did  not  provide  decisionmakers  with 
sufficient  information  on  other  initiatives  because  they  were  outside  Program 
Office  control.  These  initiatives  could  ultimately  reduce  the  volume  of  vendor 
pay  transactions  as  well  as  the  number  of  documents  to  be  managed.  Examples 
of  these  other  initiatives  were  DFAS  consolidations  and  the  use  of  credit  cards. 
Through  the  IPT  process,  the  Program  Office  assessed  the  impact  of  the  other 
initiatives  and  decided  to  limit  fielding  to  no  more  than  the  five  sites. 

Additional  sites  must  be  analyzed  and  determined  to  be  cost-effective  before 
they  can  be  fielded.  This  process  will  ensure  that  other  initiatives  affecting 
mission  need  will  be  adequately  considered. 

Analysis  of  Alternatives.  The  analysis  of  alternatives  provides  the 
decisionmaker  the  opportunity  to  evaluate  available  alternatives  to  satisfy 
mission  needs.  The  Program  Office’s  analysis  of  alternatives  documentation  in 
conjunction  with  IPT  work  supports  Program  development.  A  1993  Analysis  of 
Alternatives  and  a  1996  Functional  Economic  Analysis  were  prepared. 
Functional  economic  analyses  were  used  to  assess  cost-effectiveness  before  the 
adoption  of  DoD  5000.2-R.  Neither  document  completely  stated  the  advantages 
and  disadvantages  of  the  Program;  however,  the  documents  had  been  initiated 
before  the  DoD  5000.2-R  requirements  were  adopted.  Therefore,  the  IPTs 
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assessed  the  impact  of  other  initiatives  and  did  not  discover  any  other 
alternatives  that  had  not  been  addressed  during  the  mission  needs  statement 
review. 

The  1993  Analysis  of  Alternatives  was  limited  to  the  availability  of  technology 
at  other  organizations.  This  analysis  concluded  that  no  alternatives  existed  other 
than  to  initiate  a  new  contract  to  develop  the  solution.  The  1996  Functional 
Economic  Analysis  was  based  on  the  Omaha,  Nebraska,  operating  location. 

The  benefits  identified  by  this  analysis  were  used  as  the  basis  to  determine  the 
overall  benefits  of  the  Program,  which  were  documented  in  the  Program 
Office’s  life-cycle  cost  estimate.  Although  the  two  analyses  alone  do  not  totally 
satisfy  the  DoD  5000,2-R  requirements,  the  Program  Office  reduced  the  risk  of 
over- fielding  the  Program  by  limiting  fielding  to  five  initial  sites. 

Operational  Requirements.  The  operational  requirements  documents 
provides  the  high-level  user  requirements  of  the  selected  solution  for  the 
business  area  deficiency.  The  Program’s  operational  requirements  document 
was  prepared  and  approved  as  required.  However,  IPT  members  reported  that 
the  document  was  based  on  the  DFAS  Denver  Center  work  environment  rather 
than  the  total  expected  Increment  1 ,  Vendor  Pay,  environment.  Therefore, 
decisionmakers  did  not  know  if  the  EDM  solution  would  be  effective  for  all 
EDM  sites.  At  the  request  of  the  IPT,  the  Program  Office  performed  a  “Gap” 
analysis  of  other  sites  expected  to  receive  the  Program  to  determine  what 
modifications  were  necessary.  The  Program  Office  used  the  term  “Gap”  to 
describe  the  difference  in  requirements  from  the  test  site  to  the  proposed 
fielding  sites.  After  completing  the  Gap  analysis,  the  Program  Office  assured 
decisionmakers  that  minimal  changes  were  necessary  and  the  operational 
requirements  document  would  be  updated  to  reflect  the  needed  changes. 


Using  the  Integrated  Product  Team  Process 


The  Program  Office  prepared  for  the  Milestone  III  decision  by  using  the  IPT 
approach  to  conduct  testing,  prepare  life-cycle  documentation,  identify  and 
resolve  issues,  and  to  make  sound  and  timely  suggestions  to  facilitate  Program 
decisionmaking.  The  Program  is  the  DFAS’s  first  major  automated  information 
system  to  undergo  a  MAISRC  examination.  On  July  29,  1997,  the  Inspector 
General  formally  submitted  a  Program  IPT  assessment  to  the  Program  Office 
(Appendix  D).  On  August  6,  1997,  the  Program  Office  respond^  that  it  had 
reviewed  the  findings  and  had  already  taken  steps  to  incorporate  the  suggestions 
into  the  Program  (Appendix  E).  The  IPT  assessment  was  based  on  meetings, 
documentation  reviews,  and  test  reviews. 

Meetings.  Cost  and  Test  IPT  meetings  were  held  to  foster  open  discussions, 
raise  issues,  and  resolve  problems  early  in  the  acquisition  process.  The  teams 
consisted  of  Program  Office  and  oversight  personnel  empowered  to  act  on 
behalf  of  their  organizations.  Meetings  could  be  requested  by  any  member. 
Reasoned  disagreements  were  quickly  resolved  at  the  IPT  level.  The  IPTs  used 
continuous  up-the-line  communication  to  higher-level  staff  to  quickly  resolve 
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complex  issues,  such  as  the  handling  of  probable  DFAS  consolidations.  A 
Security  IPX  was  formed  in  August  1997  to  handle  systems  security  issues. 

Cost  Meetings.  The  Cost  IPX  consisted  of  personnel  from  the  Program 
Office  and  from  the  offices  of  the  Assistant  Secretary  of  Defense  for  Command, 
Control,  Communications,  and  Intelligence;  Director,  Program  Analysis  and 
Evaluation,  Under  Secretary  of  Defense  Comptroller;  U.S.  Army  Cost  and 
Economic  Analysis  Center,  and  Inspector  General,  DoD.  The  cost  meetings 
focused  on  the  operational  requirements  document,  life-cycle  cost  estimate,  cost 
analysis  requirements  description,  and  the  acquisition  program  baseline  issues. 
The  meetings  in  support  of  the  cost  documentation  met  regularly,  sometimes 
weekly,  for  almost  2  months  to  resolve  cost  issues.  These  contributions  were 
significant  in  achieving  timely  Program  implementation.  For  example,  the 
Program  cost  documentation  was  developed  for  17  sites.  However,  with  the 
potential  for  fewer  sites,  the  Program  Office  opted  for  a  phased  deployment 
approach,  initially  deploying  to  five  sites  with  additional  sites  to  be  evaluated  on 
a  case-by-case  basis.  Additional  information  was  needed  from  the  Program 
Office  to  identify  the  likely  cost  and  benefits  for  the  first  five  sites.  This  data 
were  used  to  determine  the  cost  effectiveness  of  the  Program  assuming  only  five 
sites  were  implemented.  The  IPX  process  saved  time  because  data  was  shared, 
eliminating  the  need  to  re-compute  all  costs  and  benefits. 

Test  Meetings.  The  Test  IPX  consisted  of  personnel  from  the  Program 
Office  and  from  the  offices  of  the  Assistant  Secretary  of  Defense  for  Command, 
Control,  Communications,  and  Intelligence;  Director,  Operational  Test  and 
Evaluation;  Director,  Test  Systems  Engineering  and  Evaluation;  Commander, 
Joint  Interoperability  Test  Command;  and  Inspector  General,  DoD.  The  test 
meetings  focused  on  the  operational  requirements  document  and  the  test  and 
evaluation  master  plan.  Although  meetings  in  support  of  testing  issues  met  less 
frequently,  major  contributions  were  still  made  toward  achieving  timely 
Program  implementation.  For  example,  although  complete  developmental  stress 
testing  had  not  been  accomplished,  the  IPX  did  not  request  the  Program  Office 
to  delay  its  initial  operational  testing.  Rather,  having  sufficient  insight  into  the 
Program,  the  IPX  concluded  that  stress  testing  could  be  addressed  later,  before 
operational  fielding  occurs.  This  allowed  the  Program  Office  to  maintain  its 
schedule.  Consideration  was  also  given  to  security  and  the  Year  2000  testing. 

Documentation  Reviews.  The  IPX  performed  documentation  reviews  as  it 
would  in  its  normal  oversight  capacity.  Through  the  IPX  process  information 
was  shared  that  facilitated  faster  resolution  of  issues. 

Command,  Control,  Communications,  and  Intelligence  Role.  The 
Office  of  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  provided  the  team  leader  for  the  Cost  and 
Test  IPXs.  The  team  leader  attended  documentation  briefings  and  ensured  that 
the  issues  raised  and  resolutions  proposed  were  well  communicated.  The  team 
leader  also  informed  the  MAISRC  of  the  Program’s  status  and  obtained 
Program  Office  feedback. 

Program  Analysis  and  Evaluation  Role.  Through  the  IPX  process. 
Program  Analysis  and  Evaluation  personnel  reviewed  the  life-cycle  cost  estimate 
and  the  cost  analysis  requirements  description.  They  also  reviewed  the 
component  cost  analysis/sufficiency  review  performed  by  the  Army  Cost  and 
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Economic  Analysis  Center.  They  found  that  the  documentation  was  in  order 
except  for  some  of  the  assumptions  requiring  further  support  and  the  effect  of 
other  initiatives  requiring  documentation.  They  also  initiated  the  process  to 
reconcile  the  difference  between  the  component  cost  analysis/sufficiency  review 
and  the  Program  Office  cost  estimate.  The  reconciliation  process  was  essential 
to  keep  the  Program  viable  as  Program  Office  personnel  were  not  experienced 
in  reconciling  the  cost  estimates.  Findings  from  the  Program  Analysis  and 
Evaluation  personnel  indicate  that  the  Program  will  be  cost-effective.  Because 
the  IPT  process  was  used,  the  issues  were  not  a  surprise  as  members  were 
updated  during  the  meetings.  As  a  result,  solutions  were  generally  discussed  at 
the  meetings. 

Test  and  Evaluation  Role.  Test  and  evaluation  personnel  from  the 
Offices  of  the  Director,  Operational  Test  and  Evaluation,  and  the  Director,  Test 
Systems  Engineering  and  Evaluation,  reviewed  the  Program’s  test  and 
evaluation  master  plan  and  the  operational  requirements  document.  Although, 
they  are  only  required  to  approve  the  test  and  evaluation  master  plan,  they 
reviewed  the  operational  requirements  document  because  the  test  plans  are  based 
on  user  requirements.  They  discovered  that  the  operational  requirements 
document  focused  on  the  Omaha,  Nebraska,  operating  location  rather  than  the 
total  Increment  1  population.  They  provided  this  information  through  the  IPT 
meetings  and  suggested  that  an  aniysis  be  performed  to  ensure  that  unique 
requirements  of  other  sites  are  considered.  Acting  on  the  information,  the 
Program  Office  performed  the  Gap  analysis  to  ensure  that  all  requirements  were 
evaluated.  Also,  during  an  IPT  meeting,  test  and  evaluation  personnel  provided 
information  supporting  the  performance  of  stress  testing. 

Inspector  General,  DoD,  Role.  The  Program  Office  invited  Inspector 
General,  DoD,  personnel  to  participate  in  the  IPT  process  to  help  facilitate 
communications  and  early  solutions  to  problems.  The  Inspector  General 
personnel  reviewed  the  life-cycle  documentation  and  the  acquisition  process  and 
shared  information  with  the  IPT  members.  Through  the  IPT,  the  Inspector 
General  provided  input  during  meetings  and  in  writing. 

Inspector  General  personnel  served  on  both  IPTs  and  kept  IPT  members  fully 
informed.  For  example.  Program  Analysis  and  Evaluation  personnel  also 
needed  to  know  the  stress  test  performance  of  the  system  to  ensure  that  the  most 
cost-effective  configuration  is  used.  Because  they  were  unable  to  attend  the 
Test  IPT  meetings,  they  would  not  have  been  aware  of  the  stress  test  issue. 

Test  Reviews.  IPT  members  also  reviewed  the  Program’s  initial  operational 
test  and  evaluation.  The  Joint  Interoperability  Test  Command  independently 
conducted  the  test.  Operational  Test  and  Evduation  personnel  approved  the  test 
after  an  IPT  meeting  in  which  test  parameters  were  questioned.  Due  to 
consolidation  activities  at  the  test  site,  the  Program  Office  proposed  that  some 
performance  requirements  be  lowered.  The  Operational  Test  and  Evaluation 
and  Inspector  General  personnel  explained  to  the  IPT  members  that  lowering  the 
requirements  could  make  the  test  results  unreliable.  The  Program  Office 
concurred  and  rescheduled  the  test  for  a  later  date  when  the  consolidation 
activities  would  not  drastically  affect  the  test.  The  test  report  from  the  Joint 
Interoperability  Test  Command  was  favorable  and  approved  by  the  Operational 
Test  and  Evaluation  personnel.  The  only  remaining  test  issues  are  stress  testing, 
system  security  testing,  and  the  Year  2000  testing.  These  solutions  were  not 
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available  during  the  initial  operational  test  and  evaluation;  however,  the 
Program  Office  has  scheduled  these  items  for  testing.  The  security  test,  the 
Year  2000  testing,  and  the  stress  test  are  scheduled  for  February  1998. 


Cost,  Funding,  and  Testing 


The  integrated  product  teams  identified  cost,  funding,  and  testing  concerns  that 
needed  to  be  resolved  before  a  deployment  decision  could  be  recommended. 

The  Program  Office  provided  documentation  effectively  resolving  those  issues. 

Cost  and  Funding.  The  IPT  ensured  that  Program  cost  was  properly  assessed 
to  reduce  the  risk  that  the  Program  was  not  cost-effective.  The  initial  Program 
was  to  be  deployed  to  17  sites.  However,  the  Program  Office  did  not  have 
sufficient  funding  for  all  sites.  Through  the  IPT  process,  the  Program  Office 
proposed  a  phased-deployment  approach  and  requested  authority  to  only  deploy 
the  first  five  sites.  The  Program  Office  satisfactorily  resolved  the  funding  issues 
for  these  five  sites. 

The  Cost  IPT  took  steps  to  ensure  that  the  component  cost  analysis/sufficiency 
review  and  Program  Office  cost  estimates  were  reconciled.  DFAS  personnel 
worked  with  the  Cost  IPT  to  reconcile  these  differences  and  establish  a  cost 
baseline  for  the  Program.  These  DFAS  personnel  also  evaluated  the  proposed 
Program  benefits  to  reduce  the  risk  that  affordability  assumptions  were 
misstated.  Once  cost  and  performance  issues  are  resolved,  the  Program  Office 
can  make  schedule  and  fielding  decisions. 

Testing.  The  Test  IPT  has  taken  steps  to  ensure  that  performance  issues  such  as 
stress,  security,  and  Year  2000  testing  will  be  accomplished.  The  Program 
Office  recommended  conducting  the  stress  testing  on  the  Increment  2  system 
configuration  because  it  represented  the  most  likely  configuration  to  be  fielded 
(for  200  or  more  users).  The  Test  IPT  members  responded  that  the  Program 
Office  recommendation  was  appropriate.  This  will  aid  in  configuration 
determinations  if  the  future  consolidations  place  more  workload  at  the  initial 
five  sites. 

A  Security  Test  Team  has  been  formed  to  ensure  that  adequate  security  is 
designed  into  the  system  and  is  tested  sufficiently.  Further,  the  IPT  members 
will  perform  or  observe  the  Year  2000  testing  in  February  1998  to  determine 
whether  the  contractor  provided  software  solution  is  adequate. 

Program  Suggestions.  IPT  members  consolidated  their  issues  and  suggestions 
so  that  the  Program  Office  could  resolve  the  issues  based  on  input  from  all  the 
members,  rather  than  from  conflicting  suggestions  of  various  oversight 
organizations.  The  Program  Office  accepted  these  suggestions  and  worked  to 
make  the  IPT  process  successful. 
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Management  Actions  Taken 


As  issues  and  concerns  were  presented  by  the  IPT  members  and  the  Inspector 
General,  DoD,  the  MAISRC  took  action  to  correct  any  deficiencies  and  mitigate 
the  acquisition  risk.  This  was  accomplished  by  giving  the  Program  Office 
limited  fielding  authority  until  the  reconciliation  of  the  cost  documentation  was 
complete.  The  MAISRC  also  identified  other  issues  in  the  acquisition  decision 
memorandum  that  the  Program  Office  needed  to  complete.  The  acquisition 
decision  memorandum,  signed  by  the  MAISRC  Chair  on  December  16,  1997, 
gave  the  Program  Office  full  Milestone  HI  approval  to  field  Increment  1  at  up 
to  five  locations  (Appendix  F).  In  addition,  before  each  site  activation  (except 
Omaha,  Nebraska)  an  agreed-upon  performance  baseline  will  be  established. 

The  Program  Office  will  be  required  to  provide  status  of  the  tasks  identified  in 
the  acquisition  decision  memorandum  through  quarterly  reporting. 

Actions  by  the  MAISRC.  On  July  29,  1997,  the  Inspector  General  provided 
the  MAISRC  action  officer  with  a  preliminary  risk  assessment  of  the  Program. 
On  August  1,  1997,  the  MAISRC  provided  limited  authority  to  field  the  system 
and  direction  to  achieve  full  fielding  authority. 

Procurement  and  Fielding  Authority.  The  Program  Office  has  been 
authorized  to  procure  long-lead  items,  conduct  site  surveys,  make  software 
modifications,  and  prepare  the  facilities.  However,  fielding  authority  was 
limited  to  the  Omaha  site  until  the  component  cost  analysis/sufficiency  reviews 
were  performed,  Program  Office  cost  estimates  were  reconciled,  and  the  first 
five  sites  were  fully  funded. 

Program  Manager  Direction.  The  program  manager  was  directed  to 
immediately  follow  the  proposed  acquisition  decision  memorandum  direction  in 
regard  to  test  and  evaluation,  security,  performance  measurement,  the  Defense 
Information  Infrastructure  Common  Operating  Environment,  the  Joint  Technical 
Architecture,  and  the  Defense  Integration  Support  Tool.  The  program  manager 
was  also  authorized  to  continue  developing  Increment  2  (Contract  Pay)  and  to 
begin  development  of  Increment  3  (Payroll  Services). 

Cost  Reconciliation  and  Funding.  The  cost  reconciliation  and  funding 
issues  were  to  be  resolved  in  60  days  from  August  1,  1997,  or  the  program 
manager  had  to  brief  the  Overarching  IPT  on  Program  status  before  proceeding 
with  further  predeployment  activities.  The  Program  Office  provided  the 
MAISRC  the  cost  reconciliation  and  funding  documentation  within  the  allotted 
time;  however,  the  Cost  IPT  required  time  to  review  the  documentation. 
Therefore,  Milestone  III  approval  was  not  granted  until  December  1997. 

Milestone  HI  Approval.  The  issues  were  resolved  and  Milestone  III  approval 
was  granted  for  up  to  five  sites  on  December  16,  1997.  The  Program  Office 
provided  the  cost  reconciliation  documentation,  funding  information,  and  a 
schedule  for  testing  that  were  reviewed  by  the  IPT  members.  The  final 
acquisition  decision  memorandum  identified  outstanding  issues  still  being 
addressed  by  the  Program  Office,  in  particular  performance  measures  baselines. 
(See  the  MAISRC’s  EDM  decision  memorandum  in  Appendix  F  for  further 
details.) 
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Conclusion 


DFAS  developed  the  required  life-cycle  documentation  and  received  a 
Milestone  III  deployment  decision  for  Increment  1,  Vendor  Pay,  of  the 
Program.  The  Program  Office’s  use  of  the  IPTs  greatly  improved 
decisionmaking  abilities  of  the  MAISRC  and  the  Program  Office.  Issues  were 
identified  early  and  the  MAISRC  and  the  Program  Office  took  action  or  planned 
to  take  agreed-upon  actions.  The  actions  taken,  based  on  the  IPT  process, 
should  r^uce  the  acquisition  risk  to  the  DoD.  This  report  did  not  make  any 
recommendations  because  actions  taken  by  management  met  the  intent  of  the 
IPT  suggestions.  However,  the  Program  Office  should  continue  to  ensure  that 
the  agreed-upon  actions  are  completed  within  the  timeframes  established  by  the 
MAISRC  and  that  all  IPT  members  are  informed  of  the  status  of  ongoing 
actions. 
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Part  II  -  Additional  Information 


Appendix  A.  Audit  Process 


Scope  and  Methodology 


Work  Performed.  The  objective  of  this  audit  was  to  assess  the  Program’s 
development  and  implementation.  Specifically,  we  reviewed  the  acquisition 
process  for  Increment  1,  Vendor  Pay.  This  increment  had  an  estimated  cost  of 
$404  million  dollars,  for  fiscal  years  1995  through  2005,  to  automate  processing 
for  13.7  million  invoices. 

The  methodology  of  the  review  included  analyses,  interviews,  and  observations 
on  the  EDM  acquisition  process.  Specifically,  we: 

o  participated  in  the  IPT  meetings; 

o  evaluated  life-cycle  documentation,  including  the  mission  needs 
statement,  operational  requirements  document,  life-cycle  cost  estimate,  cost 
analysis  requirements  description,  acquisition  program  baseline,  and  test  and 
evaluation  master  plan,  to  determine  whether  the  core  management  issues  were 
adequately  addressed; 

o  reviewed  test  plans; 

o  coordinated  with  and  interviewed  Program  Office  personnel;  and 

o  toured  the  Increment  1  pilot  facilities  at  Omaha,  Nebraska. 

Use  of  Computer-Processed  Data.  We  did  not  rely  on  computer  processed 
data  to  verify  the  reasonableness  of  the  data  provided  by  DFAS. 

Use  of  Technical  Assistance.  Operations  research  analysts  from  the 
Quantitative  Methods  Division,  Office  of  the  Inspector  General,  DoD,  provided 
assistance  in  the  review  of  EDM  cost  analysis  requirements  description  and  the 
life-cycle  cost  estimate. 

Audit  Type,  Date,  and  Standards.  We  performed  this  program  audit  from 
February  1997  through  December  1997  in  accordance  with  auditing  standards 
issued  by  the  Comptroller  General  of  the  United  States,  as  implemented  by  the 
Inspector  General,  DoD.  Accordingly,  we  included  tests  of  management 
controls  considered  necessary. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  the  DoD;  the  General  Accounting  Office,  Washington, 

DC;  and  Tecolote  Research,  Inc.,  Arlington,  Virginia.  Further  details  are 
available  on  request. 
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Management  Control  Program 


DoD  Directive  5010.38,  “Management  Control  Program,”  August  26,  1996, 
requires  DoD  organizations  to  implement  a  comprehensive  system  of 
management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  Review  of  Management  Control  Program.  We  reviewed  the 
adequacy  of  DFAS  controls  for  managing  the  acquisition  actions  and  life-cycle 
documentation  preparation  for  the  Program.  Specifically,  we  reviewed  the 
annual  statement  of  assurance  prepared  by  DFAS  headquarters  and  the  Program 
Office’s  self-evaluation.  Because  we  did  not  identify  a  material  weakness,  we 
did  not  assess  management’s  self-evaluation. 

Adequacy  of  Management  Controls.  DFAS  management  controls  were 
adequate  as  they  applied  to  the  audit  objectives. 
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Appendix  B.  Summary  of  Prior  Coverage 


Inspector  General,  DoD,  Report  No.  98-013,  "Second  User  Acceptance  Test 
of  the  Electronic  Document  Management  System  at  the  Defense  Finance 
and  Accounting  Service  Operating  Location,  Omaha,  Nebraska," 

October  24,  1997.  This  audit  evaluated  whether  system  performance  and 
control  deficiencies  identified  during  the  first  user  acceptance  test  were 
corrected.  The  auditors  found  that  DFAS  had  resolved  functional  difficulties 
reported  during  the  first  user  acceptance  test.  No  recommendations  were  made, 
and  no  management  comments  were  received. 

Inspector  General,  DoD,  Report  No.  97-050,  "Evaluation  of  Controls  Over 
Workflow  Applications  Selected  for  Electronic  Document  Management," 
December  17, 1996.  This  audit  evaluated  whether  controls  over  workflow 
applications  selected  for  EDM  were  adequate.  The  auditors  found,  through  the 
EDM  acceptance  tests,  that  the  system’s  vendor  payment  processes  and 
workflows  could  achieve  the  management  control  objectives  related  to 
completeness,  accuracy,  and  authorization  of  data  as  intended.  Still,  the  system 
failed  to  input  appropriate  security  controls  over  log-in  attempts,  meet  system 
audibility  requirements,  and  protect  secure  files.  No  recommendations  were 
made,  and  no  management  comments  were  received. 
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Acquisition  Decision  Memorandum.  Documents  the  exit  criteria  for  the 
program.  Exit  criteria  are  normally  selected  to  track  progress  in  important 
technical,  schedule,  or  management  risk  areas.  The  exit  criteria  serve  as 
“gates”  that,  when  successfully  passed  or  exited,  demonstrate  that  the  program 
is  on  track  to  achieve  its  final  program  goals  and  should  be  allowed  to  continue 
with  additional  activities  within  an  acquisition  phase  or  be  considered  for 
continuation  into  the  next  acquisition  phase. 

Acquisition  Phase.  All  tasks  and  activities  needed  to  bring  the  program  to  the 
next  major  milestone.  Phases  provide  a  logical  means  of  progressively 
translating  broadly  stated  mission  needs  into  well-defined,  system-specific 
requirements  and  ultimately  into  operationally  effective,  suitable,  and  survivable 
systems.  An  example  of  an  acquisition  phase  is  Program  Definition  and  Risk 
Reduction. 

Acquisition  Program  Baseline.  Documents  the  cost,  schedule,  and 
performance  objectives  and  thresholds  of  a  program  beginning  at  program 
initiation.  Contains  only  the  most  important  cost,  schedule,  and  performance 
parameters.  Defined  as  those  parameters  that,  if  the  thresholds  are  not  met, 
would  cause  the  milestone  decision  authority  to  require  a  reevaluation  of 
alternative  concepts  or  design  approaches.  The  values  of  the  parameters  shall 
represent  the  program  as  it  is  expected  to  be  produced  or  deployed. 

Analysis  of  Alternatives.  For  major  automated  information  systems,  this 
document  shall  be  prepared  by  the  principal  staff  assistant  for  consideration  at 
Milestone  0.  These  analyses  are  intended  to  aid  and  document  decisionmaking 
by  illuminating  the  relative  advantages  and  disadvantages  of  the  alternatives 
being  considered.  The  analysis  of  dternatives,  system  requirements,  and 
system  evaluation  measures  of  effectiveness  shall  be  clearly  linked. 

Component  Cost  Analysis/Sufficiency  Review.  The  sufficiency  review 
assessed  the  program  direction,  architecture,  schedule,  and  risk  associated  with 
the  major  cost  drivers  for  completeness,  reasonableness,  consistency,  and 
documentation.  An  independent  cost  estimate  of  high-risk  cost  elements  will 
also  be  conducted. 

Cost  Analysis  Requirements  Description.  Document  prepared  by  the  DoD 
Component  sponsoring  the  acquisition  program  that  describes  the  salient 
features  of  the  acquisition  program  and  of  the  system  to  be  used  as  the  basis  for 
the  life-cycle  cost  estimates.  For  major  automated  information  systems 
programs,  the  program  manager  shall  prepare  the  document  in  coordination  with 
the  appropriate  IPT  members. 

Integrated  Product  Team  (IPT).  Consists  of  representatives  from  the 
appropriate  functional  disciplines  working  together  to  build  successful  programs 
by  providing  input  that  enables  decisionmakers  to  make  the  right  decisions  at  the 
right  time.  Participants  function  in  a  spirit  of  teamwork  and  are  empowered  and 
authorized,  to  the  maximum  extent  possible,  to  make  commitments  for  the 
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organizations  or  functional  areas  that  they  represent.  IPTs  operate  under  the 
following  broad  principles: 

o  open  discussions  with  no  secrets; 

o  qualified,  empowered  team  members; 

o  consistent,  success-oriented,  proactive  participation; 

o  continuous  "up-the-line"  communications; 

o  reasoned  disagreement;  and 

o  issues  raised  and  resolved  early. 

Life-Cycle  Cost  Estimate.  Prepared  by  the  program  office  in  support  of 
program  initiation  (usually  Milestone  I)  and  for  all  subsequent  milestone 
reviews.  Should  be  neither  optimistic  nor  pessimistic  but  based  on  a  careful 
assessment  of  risks  and  reflecting  a  realistic  appraisal  of  the  level  of  cost  most 
likely  to  be  realized.  For  major  automated  information  system  programs,  the 
estimate  shall  include  life-cycle  benefits  in  addition  to  life-cycle  costs. 

Major  Automated  Information  Systems.  A  program  that  is  (1)  designated  by 
the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  as  a  major  system,  or  (2)  estimated  to  require  program  costs  in  any 
single  year  in  excess  of  30  million  in  FY  1996  constant  dollars,  total  program 
costs  in  excess  of  120  million  in  FY  1996  constant  dollars,  or  total  life-cycle 
costs  in  excess  of  360  million  in  FY  1996  constant  dollars.  Major  automated 
information  systems  do  not  include  highly  sensitive,  classified  programs  (as 
determined  by  the  Secret^  of  Defense).  To  determine  whether  an  automated 
information  system  is  major,  the  following  shall  be  aggregated  and  considered  a 
single  system:  (1)  the  separate  systems  that  constitute  a  multi-element  program; 
(2)  the  separate  systems  that  make  up  an  evolutionary  or  incrementally 
developed  program;  or  (3)  the  separate  systems  that  make  up  an  a 
multi-component  automated  information  system  program. 

Major  Automated  Information  Systems  Review  Council.  The  Department's 
senior-level  forum  for  advising  the  Assistant  Secret^  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  on  critical  decisions  concerning 
major  automated  information  systems  programs.  The  MAISRC  is  chaired  by 
the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence).  Principal  members  of  the  MAISRC  include  representatives  from 
the  offices  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology); 
Under  Secretary  of  Defense  (Comptroller);  the  Joint  Chiefs  of  Staff;  the 
Director,  Operational  Test  and  Evaluation;  the  Deputy  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence);  the  user 
representatives;  and  the  cognizant  senior  information  management  official(s)  or 
Component  Acquisition  Executives(s),  as  appropriate. 

Milestone.  Decision  point  that  separates  the  phases  of  an  acquisition  program. 
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Milestone  0.  Milestone  giving  approval  to  conduct  concept  studies.  For  major 
automated  information  system,  the  cognizant  principle  staff  assistant  validates 
the  mission  need  and  process  integrity  in  compliance  with  DoD  Directive 
8000.1,  “Defense  Information  Management  Program,”  October  27,  1992,  and 
the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  convenes  a  Milestone  0  MAISRC  review.  A  favorable  Milestone  0 
decision  does  not  mean  that  a  new  acquisition  program  has  been  initiated. 

Milestone  I.  Milestone  giving  approval  to  begin  a  new  acquisition  program. 
The  purpose  of  the  Milestone  I  decision  point  is  to  determine  whether  the  results 
of  Phase  0  warrant  establishing  a  new  acquisition  program  and  to  approve  entry 
into  Phase  I,  Program  Definition  and  Risk  Reduction. 

Milestone  11.  Decision  point  that  determines  whether  the  results  of  Phase  I 
warrant  continuation  of  the  program  and  whether  entry  into  Engineering  and 
Manufacturing  Development  (or  software  engineering  and  development  for  a 
software  intensive  system)  is  approved. 

Milestone  HI.  Decision  point  known  as.  Production  or  Fielding/Deployment 
Approval,  that  authorizes  entrance  into  deployment  for  an  major  automated 
information  systems  program. 

Milestone  Decision  Authority.  The  individual  designated  in  accordance  with 
criteria  established  by  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  to  approve  entry  of  an  acquisition  program 
into  the  next  phase. 

Mission  Needs  Statement.  Documents  deficiencies  in  current  capabilities  and 
opportunities  to  provide  new  capabilities  expressed  in  broad  operational  terms. 
The  mission  needs  statement  shall  identify  and  describe  the  mission  deficiency; 
discuss  the  results  of  mission  area  analysis;  describe  why  nonmateriel  changes 
(such  as  doctrine  and  tactics)  are  not  adequate  to  correct  the  deficiency;  identify 
potential  materiel  alternatives;  and  describe  key  boundary  conditions  and 
operational  environments  that  may  impact  satisfying  the  need,  such  as 
information  warfare. 

Objective  Value.  Value  that  is  desired  by  the  user  and  which  the  Program 
Office  is  attempting  to  obtain.  It  could  represent  an  operationally  meaningful, 
time-critical,  and  cost-effective  increment  above  the  threshold  for  each  program 
parameter.  Program  objectives  (parameters  and  values)  may  be  refined  based 
on  the  results  of  the  preceding  program  phase(s). 

Operational  Requirements  Document.  At  each  milestone  beginning  with 
program  initiation  (usually  Milestone  I),  contains  thresholds  and  objectives 
initially  expressed  as  measures  of  effectiveness  or  performance  and  minimum 
acceptable  requirements  for  the  proposed  concept  or  system  as  documented  by 
the  user  or  user's  representative.  Thresholds  and  objectives  in  the  operational 
requirements  document  shall  consider  the  results  of  the  analysis  of  alternatives 
and  the  impact  of  affordability  constraints. 

Phase  0.  The  objectives  of  the  Concept  Exploration  phase  are  to  define  and 
evaluate  the  feasibility  of  alternative  concepts  and  provide  a  basis  for  assessing 
the  relative  merits  (for  example,  advantages  and  disadvantages,  degree  of  risk) 
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of  these  concepts  at  the  next  milestone  decision  point.  Analysis  of  alternatives 
shall  be  used  as  appropriate  to  facilitate  comparisons  of  alternative  concepts. 

The  most  promising  system  concepts  shall  be  defined  in  terms  of  initial,  broad 
objectives  for  cost,  schedule,  performance,  software  requirements,  opportunities 
for  tradeoffs,  overall  acquisition  strategy,  and  test  and  evaluation  strategy. 

Phase  I.  The  objective  of  the  Program  Definition  and  Risk  Reduction  phase  is 
to  define  the  program,  pursuing  one  or  more  concepts,  design  approaches, 
and/or  parallel  technologies,  as  warranted.  Assessments  of  the  advantages  and 
disadvantages  of  alternative  concepts  shall  be  refined.  Prototyping, 
demonstrations,  and  early  operational  assessments  shall  be  considered  and 
included  as  necessary  to  reduce  risk  so  that  technology,  manufacturing,  and 
support  risks  are  well  in  hand  before  the  next  decision  point.  Cost  drivers, 
life-cycle  cost  estimates,  cost-performance  trades,  interoperability,  and 
acquisition  strategy  alternatives  shall  be  consider^  to  include  evolutionary  and 
incremental  software  development. 

Phase  n.  The  objectives  of  the  Engineering  and  Manufacturing  Development 
phase  are  to  translate  the  most  promising  design  approach  into  a  stable, 
interoperable,  producible,  supportable,  and  cost-effective  design;  validate  the 
manufacturing  or  production  process;  and  demonstrate  system  capabilities 
through  testing. 

Phase  EQ.  The  objectives  of  the  Production,  Fielding/Deployment,  and 
Operational  Support  phase  are  to  achieve  an  operational  capability  that  satisfies 
mission  needs.  Deficiencies  encountered  in  Developmental  Test  and  Evaluation 
and  Initial  Operational  Test  and  Evaluation  shall  be  resolved  and  fixes  verified. 
The  production  requirement  of  this  phase  does  not  apply  to  major  automated 
information  systems  acquisition  programs  or  software-intensive  systems  with  no 
developmental  hardware  components.  During  fielding/deployment  and 
throughout  operational  support,  the  potential  for  modifications  to  the 
fielded/deployed  system  continues. 

Test  and  Evaluation  Master  Plan.  Focuses  on  the  overall  structure,  major 
elements,  and  objectives  of  the  test  and  evaluation  program  consistent  with  the 
acquisition  strategy.  The  plan  should  include  sufficient  detail  to  ensure  the 
timely  availability  of  both  existing  and  planned  test  resources  required  to 
support  the  test  and  evaluation  program. 

Threshold.  The  minimum  acceptable  value  that,  in  the  user's  judgment,  is 
necessary  to  satisfy  a  need.  If  threshold  values  are  not  achieved,  program 
performance  is  seriously  degraded,  the  program  may  be  too  costly,  or  the 
program  may  no  longer  be  timely.  The  spread  between  objective  and  threshold 
values  shall  be  individually  set  for  each  program  based  on  the  characteristics  of 
the  program  such  as  maturity  and  risk. 

U.S.  Army  Cost  and  Economic  Analysis  Center.  An  independent 
organization  with  an  agreement  with  DFAS  to  perform  schedule,  cost,  economic 
analysis,  and  specialized  studies  in  support  of  DFAS. 

Year  2000  Requirement.  Addresses  corrections  in  the  hardware  and  software 
are  needed  to  ensure  continuation  of  operations  as  year  2000  approaches.  Most 
computer  hardware  and  software  use  two  digits  to  identify  the  year  instead  of 
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four;  for  instance,  1996  is  often  input,  stored,  sorted,  and  calculated  as  “96.” 
Similarly,  the  year  2000  will  be  treated  as  “00,”  the  same  designation  as  the 
year  1900.  This  will  cause  errors  in  operations  involving  sorting,  comparing, 
indexing,  and  computation  that  could  shut  down  the  system,  cause  applications 
to  operate  incorrectly,  or  worse,  slowly  corrupt  data  over  time.  Such  problems 
could  arise  without  even  being  noticed  at  first. 
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Documentation 


INSPECTOR  GENERAL 

OEPARTM6NT  OP  DEFENSC 
400  ARMY  NAVY  DRIVE 
ARUNGTON.  VIRGINIA  22202 


July  29,  1997 


MEMORANDUM  FOR  DEPUTY  DIRECTOR,  DEFENSE  FINANCE  AND  ACCOUNTING 
SERVICE,  PLANS  AND  MANAGEMENT  DEPUTATE 

SUBJECT:  Review  of  the  Milestone  III  Documentation  for  the 

Defense  Finance  and  Accounting  Service  Electronic 
Document  Management  Program 


Our  review  of  the  Electronic  Document  Management  Program 
started  on  February  19,  1997.  This  memorandum  reports  the 
initial  results  of  our  review  of  the  Milestone  III  documentation. 
We  used  guidance  contained  in  the  DoO  Regulation  5000. 2 ~R, 
"Mandatory  Procedures  for  Major  Defense  Acquisition  Programs  and 
Major  Automated  Information  System  Acquisition  Programs,"  March 
15,  1996,  to  evaluate  the  life-cycle  management  process ,  The 
review  also  considered  and  evaluated  the  use  of  tailoring  and  the 
best  business  practices,  as  needed. 

During  our  review,  we  determined  that  the  Program  Office  had 
prepared  the  required  documentation.  However,  that  documentation 
alone  did  not  provide  the  milestone  decision  authority  with 
sufficient  information  to  make  a  fully  informed  Milestone  III 
decision.  We  have  attached  our  assessment  of  the  program's  core 
acquisition  management  issues,  the  risks  involved  with 
implementing  the  current  program,  and  suggestions  that  could  be 
used  to  decrease  the  risk. 


We  suggest  that  these  points  be  addressed  to  reduce  the 
program's  risk  and  to  answer  the  core  acquisition  management 
issues.  We  plan  to  include  these  and  any  additional  Issues  which 
may  arise  in  our  draft  report.  If  there  are  any  questions, 
please  contact  Mr.  Dennis  L.  Conway,  Audit  Project  Manager,  at 
(703)  604-9158  (DConwaygDODIG . OSD , MIL) or  Mr.  Eric  Lewis  at 

(703)  604-9144  (ELewiseDODIG.OSD.MIL). 


'  F.  Jay  Lane 


Jay 

Acting  Director 
Finance  and  Accounting  Directorate 
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Rttviaw  of  tha  Blactronic  Socumant  Managamant  Progran'a 
Milaatona  ZII  Docuaantation 


Background  for  Cora  Xcguisition  Naaagamant  laauas. 

The  Executive  Summary  to  the  DoD  5000  guidance  provides 
specific  core  management  issues  that  must  be  formally  addressed 
for  every  acquisition  program.  The  milestone  decision  authority 
will  depend  on  life-cycle  management  documentation  that 
adequately  addresses  the  core  management  Issues  before  making 
program  decisions. 

We  evaluated  the  following  life-cycle  documentation  to 
determine  if  the  core  management  issues  were  adequately 
addressed: 

1.  We  evaluated  the  Mission  Needs  statement  to  determine  if  there 
was  a  validated  need  for  the  Program. 

2.  We  evaluated  the  Operational  Requirements  Document  to 
determine  what  specific  capabilities  were  necessary. 

3.  We  evaluated  the  Life-Cycle  Cost  Estimate  and  the  Cost 
Analysis  Requirements  Description  to  determine  the  cost  of  the 
Program,  if  it  was  affordable,  and  if  it  was  fully  funded, 

4.  We  evaluated  the  functional  economic  analysis  in  lieu  of  the 
analysis  of  alternatives  to  determine  if  alternative  solutions 
had  been  reviewed  and  the  reasons  for  selecting  the  solution 
being  pursued. 

5.  We  evaluated  the  Acquisition  Program  Baseline  to  determine  if 
a  program  baseline  had  been  developed. 

6.  We  reviewed  the  Test  and  Evaluation  Master  Plan  to  determine 
if  the  System  had  a  stable  design,  if  its  operational 
capability  had  been  verified,  and  if  it  was  operationally 
effective  and  suitable. 


Mission  Needs  Statement. 

Issue.  Part  2.3,  Requirements  Evolution,  states  that  DoD 
Components  shall  document  deficiencies  in  current  capabilities 
and  identify  opportunities  to  provide  new  capabilities  in  a 
mission  needs  statement.  In  the  process  of  refining 
requirements,  key  concepts  that  should  be  adhered  to  include 
keeping  all  reasonable  options  open,  facilitating  trade-offs 
throughout  the  acquisition  process,  and  avoiding  early 
commitments  to  system-specific  solutions. 

The  Electronic  Document  Management  Program's  Mission  Needs 
Statement  was  prepared  and  approved  as  required;  however,  at  that 
time  only  the  Electronic  Document  Management  solution  versus  the 
status  quo  was  considered  feasible.  Program  Office  personnel 
stated  that  they  did  not  have  sufficient  data  to  quantify  the 
effects  of  other  initiatives.  Examples  of  those  other 
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initiatives  are  the  use  of  credit  cards  and  electronic  commerce, 
which  will  reduce  the  need  for  the  Program.  Therefore,  the 
Program  Manager  and  the  Milestone  Decision  Authority  must  assess 
the  need  for  this  solution  over  time  with  other  DoD  initiatives. 
With  several  on-going  initiatives  in  the  vendor  pay  arena,  the 
Electronic  Document  Management  Program’s  payback  calculations  can 
not  be  finalized  until  this  reconciliation  is  accomplished. 

Suggested  Action.  Reconcile  the  benefits  of  other  DoD 
vendor  pay  initiatives  with  the  benefits  expected  from  the 
Electronic  Document  Management  solution. 

Analysis  of  Alternatives  and  Functional  Economic  Analysis. 

Issue.  Part  2.4,  Analysis  of  Alternatives,  states  that  an 
analysis  of  alternatives  should  aid  decisionmaking  by 
illuminating  the  relative  advantages  and  disadvantages  of  the 
alternatives  being  considered. 

A  Functional  Economic  Analysis  for  the  Electronic  Document 
Management  Program  was  prepared  in  lieu  of  an  analysis  of 
alternatives.  The  Electronic  Document  Management  Program’s 
Functional  Economic  Analysis  does  not  completely  state  the 
advantages  and  disadvantages  of  the  Program;  however,  the 
economic  analysis  was  prepared  before  the  DoO  5000  requirements 
and  was  not  intended  to  meet  those  requirements.  Further,  as 
noted  with  the  Mission  Needs  Statement,  Program  Office  personnel 
stated  they  did  not  have  sufficient  information  regarding  other 
DoD  initiatives  that  would  have  aided  in  illuminating  the 
advantages  and  disadvantages  of  the  Program.  As  a  result,  the 
most  advantageous  solution  will  not  be  known  until  the 
reconciliation  of  the  Electronic  Document  Management  Program  with 
other  programs  and  initiatives. 

suggested  Action.  The  risk  here  is  the  same  as  that  posed 
by  the  mission  need  statement  and  can  be  addressed  accordingly. 

Operational  Requirements  Document. 

Issue.  Part  2.3,  Requirements  Evolution,  states  that  at 
each  milestone,  minimum  requirements  for  the  proposed  concept  or 
system  shall  be  documented  by  the  user  or  user’s  representative. 

The  Electronic  Document  Management  Program’s  Operational 
Requirements  Document  was  prepared  and  approved  as  required; 
however,  it  did  not  completely  address  all  users*  site  and 
workload  requirements  for  Increment  1,  The  document  was  limited 
in  that  it  mainly  addressed  the  requirements  of  the  Omaha, 
Nebraska  operating  location  rather  than  all  sites  for  Increment 
I.  This  occurred  because  Program  Office  personnel  stated  that 
the  Under  Secretary  of  Defense  (Comptroller)  and  the  Director, 
Defense  Accounting  Service  had  not  determined  which  sites  would 
receive  the  System  and  consequently,  how  much  of  the  workload 
would  be  automated.  As  a  result,  some  process  and  software 
modifications  are  required  to  successfully  implement  the  Omaha 
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solution  at  other  sites.  The  Program  Office  conducted  a  "Gap** 
Analysis  to  obtain  specific  information  and  needed  process  and 
software  modifications  for  the  other  potential  sites. 

If  site  and  workload  requirements  are  not  fully  defined,  it 
is  difficult  to  accurately  determine  the  System's  cost 
effectiveness.  As  a  result,  the  specific  capabilities  necessary 
to  field  Increment  1  will  not  be  fully  known  until  fielding  and 
workload  decisions  are  made. 

Suggested  Action.  Determine  which  sites  will  be  fielded, 
how  much  of  the  workload  will  be  automated,  and  update  the  cost 
and  test  and  evaluation  plans  based  on  these  determinations. 

Life-cycle  Cost  Estimate. 

Issue.  Part  3.5.1,  Life-Cycle  Cost  Estimates,  states  that 
the  life-cycle  cost  estimates  shall  be  explicitly  based  on  the 
program  objectives,  operational  requirements,  contract 
specifications  for  the  System,  and  a  life-cycle  cost  and  benefit 
element  structure  agreed  upon  by  the  Integrated  Product  Team, 

The  estimate  should  be  comprehensive  and  identify  all  elements  of 
cost  that  would  be  needed  to  make  a  decision  on  whether  or  not  to 
proceed  with  the  development,  production,  and  operation  of  a 
system.  Further,  the  estimate  should  be  based  on  an  assessment 
of  risks  and  reflect  a  realistic  appraisal  of  the  level  of  cost 
most  likely  to  be  realized.  The  detailed  cost  information 
supporting  the  life-cycle  cost  estimate  is  contained  in  the  cost 
analysis  requirements  description. 

The  Electronic  Document  Management  Life-Cycle  Cost  Estimate 
was  prepared  as  required;  however,  some  cost  assumptions  were  not 
based  on  a  realistic  appraisal  of  the  level  of  effort  actually 
expected.  As  noted  during  the  discussion  of  the  Operational 
Requirements  Document,  the  number  of  sites  to  be  fielded  and  the 
workload  to  be  automated  is  not  fully  defined.  Other  factors 
affecting  the  cost  documentation  were; 

o  the  independent  component  cost  analysis  was 
substantially  higher  than  the  Program  Office's  estimate  which 
will  require  reconciliation  of  the  two  to  establish  a  realistic 
baseline, 

o  benefits  in  the  Program  Office's  estimate  exceed  those 
presented  in  the  Quadrennial  Defense  Review,  and 

o  funding  available  is  currently  not  sufficient  to 
successfully  accomplish  the  Program. 

Therefore,  decisionmakers  need  these  cost  issues  solved  to  reduce 
the  risk  that  they  could  make  an  improper  decision  and  so  that 
the  System  can  be  monitored  against  a  realistic  baseline. 

Suggested  Action.  Provide  cost  and  performance  metrics 
approved  by  the  Program  Analysis  and  Evaluation  personnel,  limit 
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fielding  to  a  number  of  sites  necessary  to  capture  feedback  data, 
conduct  an  in-process  review  subsequent  to  the  milestone 
decision,  and  obtain  adequate  funding  before  fielding  to 
additional  sites* 

Cost  Analysis  Requirements  Description. 

Issue*  Part  3.5.1,  Life-Cycle  Cost  Estimates,  states  that 
the  DoD  Component  sponsoring  the  acquisition  program  shall 
establish,  as  a  basis  for  the  life-cycle  cost  estimates,  a  cost 
analysis  requirements  description  of  the  salient  features  of  the 
acquisition  program  and  of  the  system  itself. 

Much  of  the  detail  and  the  assumptions  supporting  the  Life¬ 
cycle  Cost  Estimate  were  contained  in  the  Cost  Analysis 
Requirements  Description.  However,  as  indicated  on  our  review  of 
the  Life-Cycle  Cost  Estimate,  the  detailed  cost  information  was 
not  complete.  As  a  result,  a  reliable  basis  for  completing  the 
Life-Cycle  Cost  Estimate  must  be  accomplished. 

Suggested  Action.  Action  taken  on  the  Life-Cycle  Cost 
Estimate  should  be  sufficient  for  this  item. 

Acquisition  Program  Baseline. 

Issue.  Part  3. 2. 2. 2,  Acquisition  Program  Baseline  Content, 
states  that  the  acquisition  program  baseline  shall  contain  only 
the  most  Important  cost,  schedule,  and  performance  parameters. 

The  most  important  parameters  are  those  that,  if  the  thresholds 
are  not  met,  the  milestone  decision  authority  would  require  a 
reevaluation  of  alternative  concepts  or  design  approaches.  The 
values  of  the  parameters  should  represent  the  program  as  it  is 
expected  to  be  produced  or  deployed.  Therefore,  costs  shown  in 
the  life-cycle  cost  estimate,  supported  by  the  cost  analysis 
requirements  description,  should  not  differ  from  the  costs  in  the 
Acquisition  Program  Baseline. 

The  Electronic  Document  Management  Program's  Acquisition 
Program  Baseline  needed  more  specific  information  in  order  to 
establish  more  useful  and  supportable  Program  baselines.  In 
addition,  the  costs  shown  in  the  Life-Cycle  Cost  Estimate  and  the 
Cost  Analysis  Requirements  Description  needed  to  be  modified  to 
establish  a  more  realistic  baseline  since  the  number  of  sites  to 
be  fielded  were  significantly  less  than  the  Program  Office  had 
documented.  Although,  the  Life-Cycle  Cost  Estimate  and  the  Cost 
Analysis  Requirements  Description  separated  hardware  and  software 
costs  out  by  site,  the  amount  of  hardware  and  software  was  based 
on  an  estimated  workload  and  an  number  of  workers.  Additionally, 
the  benefits  are  based  on  the  same  estimated  workload  and  average 
number  of  workers  which  may  change  when  there  are  changes  in  the 
number  of  sites  actually  deployed  and  other  on-going  initiatives. 
Therefore,  there  is  a  risk  that  the  roost  accurate  and  measurable 
Acquisition  Program  Baseline  may  not  be  available  for  evaluating 
the  System's  performance,  if  specific  site  data  is  not  obtained. 
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suggastad  Action.  Establish  a  baseline  for  each  site  prior 
to  implementation  and  adjust  the  total  Program  baseline 
accordingly.  Also,  establish  criteria  to  determine  when  a 
baseline  is  breached  prior  to  implementation. 

Test  and  Evaluation  Master  Plan. 

Issue.  Part  3. 4. 1.1,  Test  and  Evaluation  Master  Plan, 
states  that  a  test  and  evaluation  master  plan  shall  be  prepared, 
and  approved  by  the  Director,  Operational  Test  and  the  Director, 
Evaluation  and  Test  Systems  Engineering  and  Evaluation. 

The  Test  and  Evaluation  Master  Plan  was  prepared  and 
approved;  however,  it  did  not  provide  assurance  that  the  test  and 
evaluation  program  was  complete.  This  occurred  because  the 
Operational  Test  and  Evaluation  and  Test  Systems  Engineering  and 
Evaluation  personnel  did  not  become  involved  with  the  Program 
until  it  was  designated  an  Acquisition  Category  lA  Program  and 
was  undergoing  unit  acceptance  testing. 

The  Test  and  Evaluation  Master  Plan  was  based  on  the 
Operational  Requirements  Document  which  contained  requirements 
specific  to  the  Omaha,  Nebraska  operating  location.  Therefore, 
as  noted  earlier  the  Operational  Requirements  Document  was 
supplemented  by  the  Gap  Analysis  as  requested  by  the  testers.  If 
the  testers  had  been  involved  in  the  Program's  development,  they 
would  have  recommended  that  developmental  stress  testing  be 
accomplished  and  would  have  required  the  initial  operational  test 
to  be  conducted  at  the  operating  site  with  the  largest  workload. 
Due  to  the  fact  that  the  Program  was  entering  user  acceptance 
testing,  the  testers  tailored  their  approach  to  realistically 
reflect  the  status  of  the  Program,  Therefore,  test  personnel 
stated  that: 

o  stress  testing  can  be  conducted  subsequent  to  the 
milestone  decision  and  before  fielding  to  other  sites,  and 

o  reviews  of  other  operational  sites  can  be  conducted 
prior  to  implementation  to  determine  the  extent  of  follow*on 
operational  testing  to  mitigate  the  risk  that  prior  developmental 
and  operational  testing  was  not  sufficient  for  the  entire 
Program , 

Suggested  Action.  Request  a  review  be  made  by  Operational 
Test  and  Evaluation  personnel  prior  to  implementation  of 
additional  sites  or  changes  in  size  or  workflow  at  existing  sites 
to  determine  the  extent  of  any  necessary  follow-on  operational 
testing.  Further,  request  that  Test  Systems  Engineering  and 
Evaluation  personnel  review  stress  testing  to  determine  the 
maximum  throughput  and  the  level  of  growth  that  the  System  can 
handle  before  it  fails. 
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DEFENSE  FINANCE  AND  ACCOUNTING  SERVICE 

1931  JEFFERSON  DAVIS  HIGHWAY 
ARLINGTON*  VA  222<40-9291 


ms  6m 


MEMORANDXJM  FOR  ACTING  DIRECTOR,  FINANCE  AND  ACCOUNTING  DIRECTORATE 
INSPECTOR  GENERAL  OF  THE  DEPARTMENT  OF  DEFENSE 

SUBJECT;  Review  of  the  Milestone  III  Documentation  for  the 
Defense  Finance  and  Accounting  Service  (DFAS) 

Electronic  Document  Management  (EDM)  Program 


Thank  you  for  the  opportunity  to  review  the  points  that  will 
be  addressed  in  your  draft  report  on  the  DFAS  EDM  Program.  Prior 
to  publishing  the  draft  report  I  want  to  take  the  opportunity  to 
discuss  my  thoughts  on  some  of  the  points  raised. 

First,  with  regard  to  the  documentation,  I  agree  that  it 
does  not  and  probably  could  never  address  all  of  the  information 
needed  to  make  a  fully  informed  Milestone  III  decision. 

Additional  information  has,  however,  been  made  available  through 
the  myriad  of  Integrated  Product  Teams  (IPT)  established  for  this 
program.  We  have  been  assured  that  the  IPT  process  was  formed  to 
raise  questions  in  an  open  forum  and  avoid  the  time 
consuming/ cumbersome  formal  "paperwork  drill”.  Having  said  this, 
however,  be  assured  where  appropriate  we  have  modified  the 
documentation  to  provide  needed  information. 

Regarding  the  Mission  Needs  Statement,  we  also  agree  that 
there  are  a  number  of  initiatives  underway  that  could  ultimately 
reduce  the  volume  of  vendor  pay  transactions  as  well  as  the 
number  of  documents  to  be  managed.  However,  because  the  affect 
of  those  initiatives  could  not  be  calculated  with  any  precision 
(in  fact,  many  are  outside  DFAS  control),  we  have  adopted  a 
deployment  strategy  that  is  both  cautious  and  flexible.  We  have 
requested  approval  to  proceed  with  the  deployment  of  only  five 
EDM  systems.  During  the  next  year  we  will  evaluate  the  impact  of 
vendor  pay  initiatives  and  assess  whether  additional  systems  are 
needed.  If  we  do  believe  that  additional  systems  are  needed  we 
will  present  this  to  the  MAISRC  separately. 

You  are  correct,  the  EDM  Operational  Requirements  Document 
was  prepared  on  the  basis  of  DFAS-Denver  work  environment.  We 
fully  acknowledge  the  need  to  make  some  modifications  to  the 
system  to  reflect  the  differing  environments  of  our  Centers.  The 
goal  was  to  build  a  common  core  system,  i.e.,  scanning,  faxing, 
archiving,  etc.,  and  permit  only  minimal  essential  modification 
to  the  index  structure/ work  flow.  Many  of  the  modifications 
emanate  from  vendor  pay  system  specific  requirements  and 
therefore  are  not  location  driven.  We  have  identified  the  vendor 
pay  systems  that  will  be  supported  by  EDM  and  have  completed  the 
“GAP”  analysis  to  determine  required  modifications.  Our  findings 
are  that  minimal  changes  will  be  needed  to  EDM. 
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The  independent  component  cost  analysis  was  initially  nigner 
than  the  Program  Office's  estimate,  however,  many  of  the 
differences  have  been  reconciled  and  it  is  my  understanding  that 
CEAC  is  modifying  their  report  to  address  the  changes.  We  will 
continue  with  this  reconciliation  until  ve  are  confident  that  ve 
have  a  solid  baseline  to  measure  both  cost  and  benefits. 

We  agree  that  stress  testing  is  essential  to  determine 
maximum  throughput  and  the  level  of  growth  that  the  system  can 
handle.  Unfortunately,  we  were  only  able  to  simulate  a  stress 
test  in  a  lab  environment  prior  to  deploying  EDM  in  Omaha.  We 
believe  ve  will  be  able  to  create  a  capability  to  further  expand 
the  test  to  simulate  nearly  500  users. 

Again,  thank  you  for  the  opportunity  to  review  your 
findings.  Please  be  assured  that  ve  have  already  taken  steps  to 
incorporate  your  recommendations  into  the  EDM  program. 


Ter/sa  K.  Walker 
l>^uty  Director  for 
Plans  and  Management 
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OFFICE  OF  THE  ASSISTANT  SECRETARY 
6000  DEFENSE  PENTAGON 
WASHINGTON.  DC  203dl-6000 


COMMAND.  CONTROL. 
COMMUNICATIONS.  AND 
INTCLUGCNCE 


MEMORANDUM  FOR  DIRECTOR,  DEFENSE  FINANCE  AND  ACCOUNTING  SERVICE 

SUBJECT:  Acquisition  Decision  Memorandum  for  Electronic  Data  Management  (EDM) 

Program 


Office  of  the  Assistant  Secretary  of  Defense  (C3I)  memorandum  of  August  1,  1997.  granted 
interim  authority  for  the  Defense  Finance  and  Accounting  Service  (DFAS)  to  obligate  funds 
regarding  EDM.  It  also  provided  acquisition  guidance  in  a  draft  Acquisition  Decision 
Memorandum  (ADM)  but  withheld  Milestone  El  approval  until  DFAS  resolved  issues  regarding 
program  cost  and  funding.  DFAS  has  satisfactorily  resolved  those  issues. 

Based  on  DFAS  resolution  of  cost  and  funding  issues  and  various  Working-Level  Integrated 
Product  Team  reviews  of  EDM.  the  anached  ADM  grants  Milestone  El  approval  to  field  EDM 
Increment  1  to  up  to  five  sites.  It  also  grants  approval  to  continue  developing  Increment  2  and  to 
begin  development  of  Increment  3. 


Acting,  MAISRC  Chair 


Attachment 


cc: 

MAISRC  Members 


o 
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ELECTRONIC  DOCUMENT  MANAGEMENT  (EDM) 
AUTOMATED  LNFORMATION  SYSTEM  PROGR.\M 
^^LESTONEIImI 

ACQUISITION  DECISION  MEMORANDUM 


Based  on  the  documentation  developed  by  the  EDM  Program  Office,  successful  Initial 
Operational  Test  and  Evaluation,  and  numerous  Working-Level  Integrated  Product  Team  (WIPT) 
meetings,  the  Major  Automated  Information  System  Review  Council  (MAISRC)  grants  the 
Defense  Finance  and  Accounting  Service  (DFAS)  the  following  approvals: 

♦  Milestone  III  approval  to  field  EDM  Increment  I  (V'endor  Pay)  to  up  to  five  sites.  For 
each  site  following  Omaha,  the  Pcrfonnancc  Measurement  WIPT  will  agree  to  a 
performance  baseline  before  site  activation. 

•  Approval  to  continue  developing  Increment  2  (Contract  Pay)  and  to  begin  development 
of  Increment  3  (Payroll  Services). 

The  MAISRC  also  directs  the  following: 

1.  Before  seeking  MAISRC  approval  to  field  Increment  1  beyond  the  first  five  site.s.  the  PM 
shall  submit,  for  Milestone  Decision  Authority  (MDA)  approval,  an  updated  Acquisition 
Program  Baseline  (APB)  reflecting  the  number  of.  and  e.stimaicd  cost  of,  the  additional 
Increment  I  sites.  If  estimated  costs  in  the  updated  .APB  exceed  the  approved  .APB  by  more 
the  10%.  the  PM  shall  work  with  the  Cost  WIPT  to  determine  whether  an  updated  Economic 
Analysis  is  required. 

2.  Follow-on  test  and  evaluation  (FOT&E)  shall  be  conducted  to  evaluate  the  additional 
Increment  1  changes  identified  in  the  EDM  Gap  .Analysis;  to  verify  interoperability  with 
CAPS,  STAR-FL  and  SAMMS;  and  to  assess  system  performance  if  the  number  of  users  at  a 
particular  site  arc  significantly  larger  than  the  Omaha  Operating  Location  (OPLOC). 

•  Before  conducting  FOTiitE.  a  stress  test  shall  be  completed  and  the  results  documented  in 
the  DT  Report  certifying  readiness  to  proceed  to  FOT&E.  The  DT  Report  must  be 
provided  to  the  Director.  Test  Systems  Engineering  and  Evaluation  before  the  Operational 
Test  Readiness  Review  for  the  FOT&E. 

•  The  PM  shall  send  a  copy  of  the  FOT&E  Independent  Evaluation  Reports  to  the 
Overarching  IPT  (OIPT)  Leader. 

3.  The  PM  shall  continue  working  with  the  Security  WIPT  to  ensure  completion  of  the  EDM 
security  accreditation,  to  ensure  the  review  of  all  EDM  security  measures  by  the  Defense 
Information  Systems  .Agency  (DISA)  and  the  Information  Assurance  Directorate 

(O AS 0(031)),  ^d  to  ensure  that  proper  security'  test  and  evaluation  is  accomplished  for 
Increment.*;  2  and  3. 
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4.  Within  90  days  of  this  ADM.  the  PM  shall  provide  the  DIPT  Leader  a  certification  that  the 
EDM  system  is  Year  2000  compliant,  or  a  status  report  describing  when  Year  2000 
compliance  will  be  achieved. 

5.  The  PM  shall  continue  to  work  with  the  performance  measurement  WIPT  to  establish  and 
implement  a  plan  for  measuring  the  improved  mission  capabilities  resulting  from  the 
deployment  of  EDM  and  to  develop  any  necessary  changes  to  the  APB. 

6.  The  PM  shall  continue  to  work  with  DISA  to  ensure  compliance  with  Defense  Information 
Infrastructure  Common  Operating  Environment  (DD7COE)  and  Joint  Technical  Architecture 
requirements,  and  to  ensure  that  the  information  regarding  EDM  in  the  Defense  Integration 
Support  Tool  is  current. 

7.  At  least  60  days  before  seeking  MAISRC  Milestone  HI  approval  for  Increments  2  and  3.  the 
PM  shall: 

•  Submit  to  the  Deputy  Director,  Space  &  Strategic  Programs  (ODPA&E)  an  updated 
Economic  Analysis  (EA)  and  a  Component  Cost  Analysis  of  the  EA.  These 
documents  shall  be  prepared  in  coordination  with  the  Cost  WIPT. 

•  Provide  to  the  OCPT  Leader  an  Operational  Requirements  Document  and  evidence  of 
OUSD(ComptroUer)  revalidation  of  the  EDM  Mission  Need  Statement. 

8.  At  least  90  days  before  seeking  MAISRC  Milestone  IE  approval  for  Increments  2  and  3,  the 
PM  shall  submit  for  OSD  approval  an  updated  TEMP  addressing  Increments  2  and  3. 


1  t  DEC  1SS7 

f .  7>v - -  - 

Margaret  E.  Myers  ®  Date 

Acting,  MAISRC  Chair 


34 


Appendix  G.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Logistics  Studies  Information  Exchange 
Director,  Test  Systems  Engineering  and  Evaluation 
Under  Secreta^  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Program  Analysis  and  Evaluation 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Assistant  Secretary  of  Defense  public  Affairs) 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 


Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Information  Systems  Agency 
Commander,  Joint  Interoperability  Test  Command 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General  National  Security  Agency 
Director,  Operational  Test  and  Evaluation 
Inspector  General,  Defense  Intelligence  Agency 
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Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division,  General 
Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional  committees 
and  subcommittees 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal  Justice, 
Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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Audit  Team  Members 


The  Finance  and  Accounting  Directorate,  Office  of  the  Assistant  Inspector 
General  for  Auditing,  DoD,  produced  this  report. 


F.  Jay  Lane 
Salvatore  D,  Guli 
Kim  A.  Caprio 
Dennis  L.  Conway 
Eric  L.  Lewis 
Suzette  L.  Luecke 
Man  W.  Lau 
Jayme  R.  Shoop 
Carolyn  A.  Gernand 
Brian  M.  Taylor 
Traci  Sadler 
Deborah  Curry 


